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I .   INTRODUCTION. 

Testing  hardware  systems  to  determine  whether  they  satisfy  operational 
specifications  is  frequently  an  expensive  and  time-consuming  task.  In 
some  circumstances,  it  is  possible  to  decrease  the  consumption  of  resources 
in  testing  by  using  sequential  test  procedures.  A  sequential  test  requires 
tnat  limits  be  placed  upon  the  desired  precision  of  the  test  (probabilities 
of  errors),  and  that  the  alternatives  under  test  (hypotheses)  be  clearly 
specified.  Such  a  test  with  two  hypotheses  is  conducted  by  taking  one 
observation  (test  value)  at  a  time  and,  after  each  observation,  making 
one  of  the  decisions 

(a)  terminate  and  reject  the  null  hypothesis; 

(b)  terminate  and  reject  the  alternate  hypothesis; 

(c)  take  another  observation. 

Since  a  sequential  procedure  requires  that  a  decision  (a  -  c  above) 
be  made  after  each  observation,  it  is  necessary  to  be  able  to  obtain  the 
observation  value  before  another  test  is  run  (assuming  it  does  not 
terminate  at  that  point).  In  the  case  of  testing  a  bombing  system  where 
several  test  runs  may  be  scheduled  in  a  relatively  short  span  of  time,  this 
means  that  the  measure  being  used  in  the  test  must  essentially  be  obtainable 
in  real  time,  without  appreciable  processing  delays.  Similarly,  some 
"field  computation",  again  without  long  delays,  may  be  required  in  order  to 
reach  one  of  the  decisions  a  -  c. 

In  what  follows  we  discuss  two  sequential  test  procedures  for  testing 
whether  a  specified  CEP  (or  less)  has  been  achieved  by  a  weapon  system 
under  given  test  conditions.  The  alternate  hypothesis  is  that  the  CEP 


is  a  second,  larger,  value  or  more.  These  values  are  obtained  from 
contract  specifications  and  operational  considerations.  One  of  the 
procedures  is  based  on  the  assumption  that  bomb  impacts  follow  a 
circular  normal  distribution,  so  that  radial  miss  distances  are 
Rayleigh  distributed.  Even  though  this  parametric  model  is  frequently 
used  in  practice,  its  adequacy  depends  upon  how  well  the  Rayleigh  fits 
the  real  situation.  There  is  mounting  evidence  [1]  that  the  Rayleigh 
model  does  not  fit  actual  drop  data  in  the  upper  tail  of  the  distri- 
bution. Hence  estimates  (such  as  CEP  =  1.1774  a)  and  tests  based  on 
this  model  may  not  perform  well,  or  even  acceptably.  In  addition,  the 
amount  of  processing  required  to  obtain  each  miss  distance  may  involve 
an  intolerable  delay  before  a  decision  is  reached  about  taking  another 
observation.  For  these  reasons, we  give  below,  only  a  mathematical 
sketch  of  the  corresponding  sequential  test,  which  we  call  the  "Rayleigh 
sequential  test." 

We  describe  the  second  test  procedure  as  "nonparametric,"  in  that  it  essen- 
tially does  not  depend  on  assumptions  about  the  distribution  of  bomb  impact 
points  about  the  target.  This  procedure  utilizes  the  observed  proportion 
of  impacts  that  fall  within  the  specified  CEP.  It  is  assumed  that 
judgement  about  whether  a  given  bomb  fell  within  the  specified  CEP  can  be 
made  without  significant  processing  delay,  so  that  it  will  be  known 
shortly  after  a  given  bomb  impact  is  observed,  whether  another  test  run 
is  required.  This  sequential  procedure  is  mathematically  equivalent 
to  a  well  known  procedure  frequently  used  in  reliability  testing  and 
quality  assurance  applications;  in  what  follows  we  call  it  the  "sample 
proportion  test." 
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1 1 .  DESCRIPTION  OF  TESTS. 

We  use  the  Sequential  Probability  Ratio  Test  (SPRT)  developed  by 

Wald  [2].  The  test  may  be  described  in  terms  of  a  random  walk  tt 

and  bounds  B  <  A,  such  that  the  test  continues  so  lonq  as  tt 

a     n 

remains  between  B  and  A.  The  statistic  tt   is  based  on  the  likelihood 
ratio,  which  in  turn  depends  upon  the  statistical  model  (in  our  case, 
Rayleigh  or  binomial)  and  the  CEP  values  specified  under  the  hypotheses. 
We  shall  employ  the  usual  approximations  to  the  stopping  bounds  B  and 
A, 

B=1A;  A  =  M, 

1-a        a 

wnere  a  and  3  are  the  type  I  and  type  II  error  rates,  respectively. 
There  is  strong  evidence  [3]  that  these  approximations  are  quite  adequate, 
except  possibly  for  very   small  a  and  6  . 

A.  Rayleigh  Test.  It  is  convenient  to  state  the  test  in  terms  of 

2  2 

CEP  and  squared  radial  miss  distances,  R  .  Since  the  original  quantities 

are   nonnegative,  this  test  is  equivalent  to  what  would  be  obtained  with 

CEP  and  R. .  Now  assume  the  Rayleigh  model  for  R,  so  that 

2 
R2~  FR2(y)  =  l-e"y/2°  ;   y  >  0  ,  (1) 

2 
an  exponential  distribution  with  parameter  A  =  l/2a  .  Hypothesizing 

CEP  =  C  is  equivalent  to  setting  X  =  a/C^  (say). 

Let  Cn  denote  the  squared  CEP  specified  and  C,  denote  the 
squared  CEP  alternative,  where  (CQ,C-,)  is  an  indifference  zone.  Presumably 
we  are  indifferent  to  which  hypothesis  is  rejected  when  in  fact  the  true 
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2 
CEP  squared  is  between  CQ  and  C]  .   In  order  to  test  CEP  £  CQ  against 

CEP2  :>  C-,  ,  we  test  the  simple  hypotheses 

HQ  :  CEP2  =  CQ  vs  H]  :  CEP2  =  C]  (2) 


The  SPRT  continues  so  long  as 

n  f(y1  ;  C,) 

B  <  *  f(y  •  C 1  <  A  '  (3) 

i=l  nyi  '  V 

where  f(y.  ;  C)  ,  the  density  corresponding  to  (1)  with  — j  =  a/C.  , 
i    J  2(/     J 

is 

f(y.  ;  Z.)   =   ^-exp(-  a./C^)  .  (4) 

Taking  logs  in  expression  (3),  with  the  explicit  expression  (4)  for  f, 
the  test  may  be  described  as  follows: 

SEQUENTIAL  RAYLEIGH  TEST:  In  order  to  sequentially  test  hypotheses  (2)  with 
approximate  size  a  and  power  1-3  when  sampling  from  density  (4),  con- 
tinue  taking  observations  R   so  long  as  their  sum  (the  sum  of  the  first 
n  observations)  falls  between  the  acceptance  numbers  a   and  the  rejection 


numbers  r  ,  where 


n   2 


Terminate  and  accept  Hn  whenever   V  R  <.   a  ;  terminate  and  reject  hL 

r         U  L?rn  u 


n       2  r=l 

whenever       T     R     :>  r     . 
^t     r         n 
r=l 
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The  operating  characteristic  L  of  this  test  is  approximately 

L(C)  =  P[Test  accepts  HQ  |  CEP2  =  C]  «  (Ah(C)  -  l)/(Ah(C)  -  Bh(C))  , 
where  h(C)  is  any  (positive)  solution  to  the  equation 

/r  \h(c) 
C  = 


h(C)  [i  .  1  ] 

Ll   Lo 


Let  N  denote  the  (random)  sample  size  at  termination.  The  average 

2 
sample  size  required  for  termination  when  CEP  =  C  is  approximately 


9       [l-L(C)]ln  ±£ ■  +  UOln^rf- 
E(N  |  CEP^  =  C)  k        c 


ln_£  +  c[l  .  1] 

B.  Sample  Proportion  Test.  In  order  to  test  the  hypotheses  (2) 
without  making  strong  assumptions  about  the  distribution  of  impact  points 
about  the  target,  one  may  transform  each  data  point  by  an  indicator  function 
which  gives  "hit-miss"  data.  Suppose,  then,  that  X,  is  1  if  R.  ^  /C7  ; 
X.  =  0  otherwise.  Assuming  the  miss  distances  R, ,  R?,  ...  are  a  random 
sample  from  an  (unknown)  continuous  distribution,  X,,  X~,  ...  are  I  ID 

Bernoulli  random  variables  under  hL  and  H  .  For  these  transformed 

2  ' 

variables  the  null  hypothesis  hL  :  CEP  ^  CQ  is  equivalent  to  hL  :  p  ^  1/2  , 

where  p  is  the  "success"  parameter,  p  =  P[X  =1].  We  also  need  to 

determine  limits  on  p  under  H  :  CEP  s  C, .  This  requires  a  model  which 

r        a         1        ^ 
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relates  P[R.  <;  /CT  ]  to  C-,  ,  through  the  distribution  of  R. .  For  this 
purpose,  we  assume  a  Rayleigh  formula.  However,  this  is  not  a  strong 
assumption  nere   (in  contrast  to  the  parametric  model  discussed  above), 
because  we  do  not  rely  on  the  formula  for  values  far  from  the  hypothesized 
medians  of  the  distributions.  It  has  been  noted  above  that  the  Rayleigh  model 
gives  adequate  fit  for  these  values;  it  is  the  tai 1  behavior  of  the  Ray- 
leigh that  seems  to  invalidate  its  use  for  many  CEP  estimation  and  testing 
problems.  Since  we  are  using  the  Rayleigh  formula  for  moderate  values, 
and  in  fact  using  it  only  to  assess  P[X.  =  1   CEP  =  C-,],  we  feel  this 
is  not  a  serious  assumption  and  that  it  is  not  misleading  to  refer  to  the 

present  test  as  a  "nonparametric"  test.  Under  this  assumption  and  with 

2 
the  transformed  variables  X.,  the  alternate  hypothesis  H  :  CEP  ^  C, 

i 

is  equivalent  to  H  :  p  ^  p-,  ,  whe 

a       i 


re 


P]  =  P[X.  =  1  |  CEP2  =  C^  =  P[R2  s  CQ  |  CEP2  =  C^ 


K  °) 


=  1  -  exp 


-c0/c1 

=  1  -  2  U  ' 


The  likelihood  ratio  in  this  case  is 

n 

z  xi 

n     f(Xi    ;Pl)     _/   Pl\1=1 
i!1   f(Xi    ;  1/2)  -\V^j  [20-P,)]     . 

n 
which  remains  between  B  and  A  so  long  as  £  X.  is  between 

i=l  1 

an  =  (l«A+n(cC°-l)  B  )/,„(2VCl.,j  (4) 
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and 

the  "acceptance  number"  and  "rejection  number,"  respectively.  Note 
that  the  test  is  dependent  upon  C.   and  C-,  only  through  their  ratio, 

c0/c, . 

We  can  now  state  our  "nonparametric"  sequential  ChP  test  as  follows: 
SAMPLE  PROPORTION  TEST.  In  order  to  sequentially  test  hypotheses 
{'l)   with  approximate  size  a  and  power  1-3  ,  make  independent  observa- 
tions of  whether  each  bomb  radial  misdistance  is  within  /CT  (so  X.  =  1) 


or  is  not  (so  X.  =  0).  Continue  taking  observations  X,  so  long  as 

a  <  Z  X.  <  r   where  a   and  r   are  given  in  (4).  Terminate  and  accept 
n   -j  =  i  l    n        n       n  r 

n  n 

Hn  whenever  Z,  X,  ^  a„  ;  terminate  and  reject  Hn  whenever  Z  X.  ^  r  . 
u         -j  =  |in  u         -j  =  ]  l    n 

The  operating  characteristic  L  of  this  test  is  approximately 
L(C)  -  P(test  accepts  HQ  |  CEP2  =  C)  «  (Ah(C)  -  l)/(Ah(C)  -  Bh(C))  , 
where  h(C)  is  any  positive  solution  to  the  equation 


C  =  -  CQ  a   /  In(l-J)  , 


where 

J  = 


,  .  ,h(c)(2-Vci)h(c) 

zn(C)L2-C0/CMC)  -   2h(C)(2"C°/Cl)  hlC> 
The  expected  sample  size  N  required  for  termination  of  this  test  is 
approximately 


7  - 


0  [l-L(C)]ln  -^  +  L(C)ln  y^- 

2  n\  VJ    a     v  '   1-a 


U^UJ^l   2^.  U£o/Cl 


E(N   CEP^  =  C)  - 


The  exact  operating  characteristic  and  mean  sample  size  functions  do  not 
differ  substantially  from  the  above  approximations  when  a  and  6  are 
not  below  .1  [3].  The  variance  in  sample  size,  V(N   CEP  =  C) 
apparently  does  not  admit  such  good,  simple  approximations.  However 
there  is  empirical  evidence  [3]  that  the  standard  deviation  of  N  tends 
to  be  of  the  order  of  magnitude  of  the  mean  of  N.  This  means  that, 
occasionally,  sampling  may  continue  substantially  beyond  E(N).  On  the 
other  hand,  it  is  known  [4]  that  the  SPRT  achieves  the  smallest  average 
sample  size  of  any  test  of  these  hypotheses  operating  with  the  same  error 
rates  a  and  3. 
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III.  PERFORMING  THE  TESTS. 

We  remarked  above  that,  for  the  proportion  test,  the  acceptance 
number  and  rejection  number  depend  on  CQ  and  C,  only  through  the  ratio 
Cq/C-,  .  This  implies  that  performance  of  the  proportion  test,  as  well  as 
its  operating  characteristic  and  average  sample  size  functions,  depend 

upon  the  hypothesized  values  only  through  CQ/C, .  A  similar  remark  can 

2 
be  made  for  the  Rayleigh  test  inasmuch  as  CEP  may  be  viewed  as  a 

2 
scale  parameter  for  the  distribution  of  squared  radial  miss  distance  R 

This  means  that  one  can,  in  effect,  measure  the  squared  miss  distances 

2 
R. ,  and  the  hypothesized  values  CQ  and  C-,  in  "C^-units".  In  such 

2  2    2 

a  case,  measurements  R.  are  converted  to  T.  =  R^/Cq  ,  CQ  is  taken 

to  be  Cq  =  Cq/Cq  =  1  ,  and  C-,  is  converted  to  C-,  =  C,/Cq.  Such  a 

point  of  view  allows  one  to  use  tabulations  of  a  number  of  "standard" 

tests  in  order  to  perform  tests  of  a  much  wider  variety  of  hypotheses. 

To  illustrate  this  point,  we  consider  one  example  of  each  of  the 
tests  described  above.  For  the  case  0-,/Cq  =  2  and  a  =  3  =  .1  » 
we  present  graphical  aids  which  would  simplify  performance  of  the  tests, 
as  well  as  graphs  of  the  operating  characteristic  curves  and  expected 
sample  size  curves.  For  the  Rayleigh  case,  we  also  show  a  sample  size 
variance  curve  obtained  through  simulation. 

A.  Example  Sequential  Rayleigh  Test.  Suppose  we  wish  to  test 
HQ  :  CEP2  <  1000  vs.  H  :  CEP2  >  2000  with  a  =  3  =  .10  .  With 
C,/CQ  =  2  ,  we  may  take  CQ  =  1  and  C,  =  2  as  discussed  above,  with 
the  agreement  that  the  squared  radial  miss  distances  actually  observed 
are  transformed  by  dividing  by  1000  before  computing  the  test  statistic 
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Z  R.  .  Equivalently,  (and  more  simply),  we  can  compute  the  test 
i=l   • 

statistic  with  the  "raw"  data  actually  observed  and  transform  the  acceptance 

and  rejection  numbers  by  multiplying  by  1000.  Thus  the  test  continues  so 

2 
long  as  E  R.  is  between 

a  =1000  [net  -  lng]  /  Hpl   =  2000n  -  6339.85 
r  =1000  [na  +  ln9]  /  H&   =  2000n  +  6339.85 


Plotted  as  functions  of  n  ,  these  bounds  are  parallel  lines  (as  shown 

below)  having  slope  2000  and  intercepts  +  6339.85, 

2 

ZR 


Accept  HQ 


FIGURE  1 


A  hypothetical  sequence  of  observed  values  of  E  R.  is  shown  plotted 
against  n  (shown  as  dots  in  the  figure);  that  is,  the  points  (1 ,  y-, )  , 
(2,  y,  +  y2)  ,...,  (6,  E  y . )  .  In  performing  the  test,  these  points 
would  have  been  sequentially  plotted  on  a  previously  prepared  "control 

chart"  with  the  parallel  a   and  r   lines.   Note  that  the  hypothetical 

2 
test  terminated  with  the  6th  observation,  since  the  walk  (n,  Z  R . ) 

left  the  continuation  region  for  the  first  time  at  n  =  6;  the  terminal 

decision  was  to  accept  1-L  with  the  example  shown.  The  operating 

characteristic  curve,  expected  sample  size  curve  and  variance  in  sample 

size  curve  for  this  test  are  shown  in  figure  2,  as  functions  of  C/CQ  . 
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Note  that  the  operating  characteristic  curve  passes  through  (or  nearly 
so)  the  points  (a,  L(l))  and  (1-3,  L(2)),  as  should  be  the  case.*  Note 
also  that  mean  sample  size  is  largest  for  C/Cn  values  between  1  and 
2,  again  as  should  be  the  case,  because  such  values  would  typically  give 
samples  which  are  not  more  compatible  with  one  of  the  hypotheses  than  the  other 
so  no  early  decision  is  reached.  Similar  observation  can  be  made  with 
respect  to  variance  in  sample  size. 

B.  Example  Sequential  Proportion  Test.  The  sequential  test  based 
upon  "hit  -  miss"  data  can  also  be  conveniently  performed  with  a  control 
chart.  With  the  values  of  (a,  3)  CQ  and  C,  proposed  in  the  preceding 
example,  we  have  for  this  test  the  hypotheses  HQ  :  p  ^  .5000  vs.  H  :  p  <  p,  = 
1  -  1  /  /Z  =  .2929.  The  acceptance  and  rejection  numbers  in  this  case  are 

a  =  [-ln9  -  na/2]  /  ln[/2"  -  1]  =  .3932n  +  2.4929 
r  =  [ln9  -  na/2]  /  ln[/2"  -  1]  =  .3932n  -  2.4929 

which  as  functions  of  n  are  lines  with  slope  .3932  and  intercepts  + 
2.4929.  In  this  case  the  control  chart  is  as  shown  in  figure  3. 


Re^ct  "0 


Figure  3 


*The  0C  curve  does  not  pass  precisely  through  these  points  because  of 

the  Wald  approximation  of  B  and  A  .  These  curves  are  based  on  empirical 

values  determined  through  simulation  by  Gavlak  [3]. 
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In  this  figure  we  show  a  hypothetical  sequence  of  values  (n,  .Z  X.) 

which  led  to  acceptance  of  HQ  at  the  tenth  observation.  The  O.C. 

curve  and  expected  sample  size  curve  are  shown  in  figure  4,  as  functions 

of  C/Cq.  These  curves  are  based  on  Wald  approximations. 

It  is  interesting  to  compare  the  O.C.  curves  and  expected  sample  size 

(E(N))  curves  for  the  Rayleigh  and  proportion  tests,  for  the  example  case. 

Note  that  the  O.C.  curves  are  quite  similar,  but  that  the  E(N)  curve 

for  the  proportion  test  is  roughly  twice  as  high  as  that  for  the  Rayleigh 

test.  For  example  under  HQ  the  sequential  proportion  test  has  EQ(N)  •*  18 

(a  nonsequential  proportion  test  requires  n  =  97),  whereas  the  Rayleigh 

has  EQ(N)  =  8.8  ;  under  the  alternative  hypothesis  the  proportion  test 

has  E  (N)  k  20  whereas  the  Rayleigh  has  E  (N)  =  10  ;  the  maximal 
a  a 

expected  sample  size  for  the  proportion  test  is  approximately  26  at  C  =  1 -4Cn 
whereas  it  is  15  at  C  =  1 . 5CQ  for  the  Rayleigh.  One  might  interpret  these 
comparisons  as  the  "cost"  of  using  the'nonparametric  procedure  if  the  para- 
metric procedure  were  appropriate.  Since  these  relationships  are  probably 
typical,  we  may  assert  that  the "nonparametric" procedure  gives  protection 
against  invalid  testing  due  to  departures  from  the  parametric  assumption,  at 
the  expense  of  additional  observations  (on  the  average). 

As  is  typically  true  of  tests  of  this  type,  the  expected  sample  sizes 
will  generally  decrease  with  any  of  the  following: 

a.  increase  in  a 

b.  increase  in  3 

c.  increase  in  the  ratio  C,/C0  . 


A  decrease  in  E(N)  for  true  C  values  not  near  the  mid  point  of  (CQ,C,) 
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IV.   CONCLUSIONS. 

The  tests  discussed  above  should  prove  useful  in  testing  whether 
CEP  specifications  are  being  met  under  various  test  conditions.  The 
"nonparametric" test,  the  sequential  proportion  test,  requires  more 
observations  (on  the  average)  than  its  parametric  competitor,  but  is 
not  invalidated  by  departures  from  the  parametric  assumptions  required 
by  the  latter.   Indeed,  with  the  sequential  proportion  test,  one  does 
not  have  to  be  concerned  with  most  of  the  "model  validation"  problems 
usually  encountered  in  performing  a  statistical  analysis.   (All  that  is 
required  is  that  the  observations  be  independent,  and  that  the  test 
conditions  be  reasonably  homogeneous. 

Research  is  currently  being  conducted  on  a  third  sequential  test 
which  may  prove  useful  in  the  context  considered  here.  This  test  will 
be  based  on  sequentially  observed  sample  medians,  with  the  null  hypothesis 
being  rejected  when  the  sample  median  falls  too  far  from  /CT  .  A  second 
area  of  research  effort  is  concerned  with  methodology  for  combining 
results  observed  under  the  various  test  conditions  into  an  overall  test  of 
performance.  Results  of  these  efforts  will  be  reported  in  a  future  paper. 
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